Variance reduction in feature hashing using MLE and control variate method

The feature hashing algorithm introduced by Weinberger et al. (2009) is a popular dimensionality reduction that compresses high dimensional data points into low closely approximate the pairwise inner product. This has been used in many fundamental machine learning applications such as model compression (Chen 2015), spam classification (Weinberger 2009), compressing text classifiers (Joulin 2016), large scale image (Mensink 2012). However, limitation of this approach variance its estimator for product tends to be small values reduced dimensions, making estimate less reliable. We address challenge and suggest two simple practical solutions work. Our relies on control variate (CV) maximum likelihood (MLE), which are techniques statistics. show these methods lead significant similarity estimation. give theoretical bounds same complement it via extensive experiments synthetic real-world datasets. Given simplicity effectiveness our approach, we hope can adapted practice.